The invention relates to the field of communication methods. In particular, the invention relates to software for identifying sketch entities from sketch activity and for correlating media information to these sketch entities.
Short-term communication between two or more distant people is typically performed on the audio level. A variety of telephone systems provide the proper tools for that type of communication.
To exchange more specific information, communication solely on the audio level becomes often unsatisfactory. Visual information in the form of graphics, pictures, sketches and the like are used to aid the information exchange.
In meetings, where people are physically together, visual information is shared by making it simultaneously visible to all participants. In addition, the participants are able to express themselves by using gestures or by drawing sketches.
Devices have been developed that provide shared visual information correlated to audio information. The goal of such devices is to enable people in distant locations, to communicate verbally and share visual information at the same time. The limited transmission capacity of public switched telephone networks (PSTN) reduces the feasibility of simultaneous audio and visual information exchange. The exchange of more detailed visual information like for instance pictures, graphics or sketches is not possible with such systems. Real time multi media communication devices (RTMMCD) that use the PSTN typically provide only a low resolution screen that is able to capture and transmit facial expression of a participant.
One major problem of multi media communication is the large variation in the data amount of the transmitted audio and visual information. These discrepancies occur because visual and verbal information are typically correlated in an information exchange event. As a result, high data amount of simultaneous audio and visual information intent to exceed the transmission capacities of the communication infrastructure. Since a signal distributed between a number of users via a PSTN can carry only a certain amount of information within a given time period, the transmission of visual and verbal information needs to be buffered to allow the transmission of more sophisticated visual information.
The buffering of the information is typically accomplished by independently saving audio information and/or video information. This buffering is accomplished temporally and/or permanently, at the location where the information is created and/or at a remote location. In a following step, the correlated information are transmitted chronologically with certain user definable parameter.
U.S. Pat. No. 4,656,654 to Dumas discloses a computer-assisted graphic teleconferencing method and apparatus that is designed for use with the PSTN. The method and apparatus described in the patent work according to the principles described in the paragraph above. The main disadvantage of this invention is that graphics and voice can be communicated only alternatingly. A simultaneous distribution of a sketching activity with the contemporaneous explanatory verbal information is not possible with this invention. In addition, the invention is not usable in combination with the Internet since no distribution system is described that may be implemented in a web page.
U.S. Pat. No. 5,801,757 to Saulsbury discloses an interactive communication device that allows simultaneous sending and receiving of audio and graphic information via a PSTN. The device uses techniques for compression, merging and coding of signals to accomplish the transmission. The patented device further uses techniques for decompressing, separating and decoding of signals to recreate the audio and graphic signals in their original form at the location of a receiver. The patented device is placed between the telephone line and the PC.
The device provides a possibility for simultaneous exchange of audio and graphical information. The main shortcoming of the device is that it needs to be physically installed in combination with a software program, which may result in problems of compatibility with existing hardware. Furthermore, it is not possible to communicate audio-graphically with a person that is not in possession of the device. The invention is also not usable in combination with the Internet since no distribution system is described that may be implemented in a web page.
U.S. Pat. No. 5,832,065 to Bannister et al. discloses a synchronous voice/data message system that allows the exchange of audio-graphic messages between specific portable communication devices also via a PSTN. The message system provides a replay function to display the creation process of the graphical information. In addition, the message system simultaneously replays the correlated verbal information. The chronological audio graphic information can be replayed at varying speeds. Unfortunately, the message system is one directional and chronological. It does not afford a recipient the option to selectively access segments of the chronologically retrieved message. It is not possible to communicate audio-graphically with a person that is not in possession of the portable communication device. Further, the invention is not usable in combination with the Internet since no distribution system is described that may be implemented in a web page.
U.S. Pat. No. 5,915,003 to Bremer et al. discloses a sketching unit for transmission of sketches and notes over normal telephone lines. The teaching of the patent is similar to that of Saulsbury. It utilizes in addition a specific sketching unit that allows creating and/or displaying graphic information. The patent further discloses a technique for a multiplexed transmission via a device that is switched between the telephone line and a computer. It is not possible to communicate audio-graphically with a person that is not in possession of the device. The invention is also not usable in combination with the Internet since no distribution system is described that may be implemented in a web page.
A communication medium that is gaining more and more significance is the Internet. A number of software products and web pages provide users possibilities to exchange audio and/or graphical information for the purpose of real time collaboration.
For instance, the RealityWave Inc. discloses on their web page www.realitywave.com a software product called VizStream that allows to create 3D graphics that can be embedded within a web page and accessed by the client. Even though the software provides an enhanced display technique, it limits the client to view a prepared information. Bi-directional information exchange on the basis of a common document is not possible with that technique. Further, Vizstream provides only the display of 3D models without any additional media information like, for instance, audio, video, or graphics.
On the web page www.solidworks.com a software program called xe2x80x9ceDrawingxe2x80x9d is presented, which allows to generate self extracting files that can be attached to emails. The self extracting files unfold into an interactive screen where 2D mechanical drawings can be viewed together with remarks and any other text or graphical information necessary to make the drawing understandable. eDrawing is also one-directional, which means that the client cannot add on his side to the contents of the information. Further, eDrawing provides no possibility to add verbal information to the drawing.
On the web page www.bluelineonline.com web site based service programs are introduced by the names of xe2x80x9cProjectNetxe2x80x9d, xe2x80x9cProjectNet LTxe2x80x9d and xe2x80x9cProjectNet EPSxe2x80x9d. Among other services, the programs provide a number of clients engaged into the same project with the possibility to review simultaneously technical drawings. In addition, the programs enable the clients to add predetermined graphical symbols and explanatory text to the drawing. This added information is distributed to all other clients for the purpose of review.
Even though the programs greatly improve real time collaboration they restrict the clients to the use of predetermined graphical symbols together with written text.
Sophisticated information elements within a single displayed image and/or in a chronological context cannot be captured directly by the programs. In addition, the information is restricted to visual information that need to be manually added. No possibility to incorporate audio information is provided.
Therefore, there exists a need for a method and system to allow two or more person to communicate audio graphically without significant time delay, without the need for specific equipment and without limitations imposed by the transmission capacity of the available communication infrastructure. The present invention addresses this need.
It is a primary object of the present invention to provide a method that allows a number of clients to freely document graphical information together with multi media information like, for instance, audio and/or video information.
It is a further object of the present invention to provide a method that captures the correlation between graphical and other multi media information for a chronological presentation at client locations.
It is another object of the present invention to provide a method that presents the captured graphical information and the correlated multi media information in a mode such that the client can select any graphical information element individually; by making the selection, the software should simultaneously replay the correlated multi media information element.
It is another object of the present invention to provide a method that allows to essentially simultaneously exchange information added to a graphical and multi media document between a number of clients.
In addition, it is an object of the present invention to provide a method that keeps a number of graphical and multi media documents independently available for review and modification by a number of clients.
Finally, it is an object of the present invention to provide the method in a form that allows it to be accessed by a number of clients via the internet and/or internet related services like for instance emailing.
The present invention introduces a software program that allows clients to exchange,graphical information together with correlated multi media information. Correlated multi media information is primary verbal information and secondary video information.
The software program provides the exchange in a quasi simultaneous mode. Since real time information exchange is influenced by the transmission capacity of the communication infrastructure the software program provides a script log for each client and project. In the script log all events during the creation of a graphical and multi media document are temporally correlated. Further, the software program recognizes free created graphical entities by capturing the activities of input devices. An input device is, for instance, a mouse, a digitizer tablet, or a pointer of a touch screen.
The creation of a graphical entity begins typically with an initiation event performed by the user. This initiation event is performed with the down click of a mouse button or by bringing a pointer into contact with a touch screen. The creation of a graphical entity ends typically with an termination event performed by the user. This termination event is performed, for instance, with the release of the down held mouse button. The period between the initiation event and the termination event define the temporal boundary condition to combine a number of drawn line segments into a sketch entity. This definition system is applied in a basic and an advanced form with the result of sketch entities with varying complexities.
A video input device as for instance a video camera may capture in addition visual information correlated to the graphical information. The visual information is primarily provided by the user and may, for instance, be the facial expressions and gestures of the user or any other visual information correlated to the creation of the graphical information.
An audio input device as, for instance, a microphone captures audio information correlated to the graphical information. The audio information is primarily provided by the user in the form of verbal information.
Graphical, visual and audio information are time stamped, captured and stored. In the preferred embodiment of the invention, the storing is performed in the form of a dynamic script log on a direct-access storing medium like, for instance, a disk drive or the read active memory (RAM) of the users computer. As a result, the correlation of graphical, visual and audio information can be reconstructed.
Since verbal information is not necessarily synchronous with the period of each correlated initiation action, the invention recognizes bulks of audio information and correlates them to the corresponding sketch entities.
The Internet allows each individual user to retrieve and transmit information independent of the capacity of the communication infrastructure. In such a buffered transmission mode, the transmission capacity of the communication infrastructure solely influences the waiting time to send and/or retrieve the information. In correspondence with this buffered transmission mode, the present invention provides a buffered transmission mode, during which the created script log is transmitted to a central server and eventually broadcasted in a quasi real time mode that corresponds to the transmission capacity of the communication infrastructure.
The Internet also allows streaming information transmission during which the information is presented as it is received and/or created. Streaming transmission is utilized for instance for so-called chat rooms or streaming video. With increasing transmission capacity of the communication infrastructure, on which the Internet is based, streaming data transmission via the Internet becomes increasingly relevant. The present invention provides a streaming transmission mode, during which data is distributed between the number of participants as it is created.
The preferred system architecture of the present invention consists of one or more main server stations that can be accessed by the clients via a web page. Such a web page operates as a broadcasting site that receives and redistributes all information from the individual clients and/or participants. The web page provides an interactive graphical interface, in which the clients can replay, view, edit and/or create sketch information.
During the replay mode the creation process of a document can be replayed on the interactive graphical interface in a real time mode and/or in a temporally altered mode. Correlated audio and/or video information is replayed simultaneously.
During the viewing mode, individual sketch entities can be selected and the correlated audio and/or video information is replayed. Since sketch entities do not necessarily have media information associated with them, the invention provides an optional highlight mode. The highlight mode allows the reviewing client to visually recognize additional media information correlated to individual sketch entities.
During the editing mode, the client can add sketch information to a retrieved document. At the same time, the client can record audio and/or video information to contribute to collaborative creation of a document. The invention provides a selectable graphical vocabulary like, for instance, line fonts or colors that can be assigned to individual clients. As a result, each contribution can be correlated to its creator. The invention provides the possibility to either broadcast the collaborative editing in a quasi real time mode respectively a streamed real time mode and/or an off-time mode. During the off-time mode, individual participants may contribute at any time to the creation of the document. The invention provides thereby an information system that informs other participants about an update of a document under collaboration.
In addition, the interactive graphical interface a background display mode, during which graphical and/or pictographic images may be displayed. In doing so, clients are able to incorporate previously created documents like, for instance, blueprints, photographs, maps, snapshots and/or video frames.
In an alternate embodiment, a client may be provided with a software program of the present invention in the form of a self-extracting email message, and/or an installable program downloaded from a web page. The installable program may also be retrieved from a storage medium like, for instance, a Floppy Disk or a Compact Disk. As a result, the client is able to perform all operations of the present invention on his/her own computer without being connected to the Internet. In this embodiment, each client occasionally exchanges information either with a server station or directly with other clients to exchange all updates.
The present invention may further be part of an operating system that operates a computer and/or a communication device like, for instance, a cellular phone. The operating system may include the operation of a communication network.
The system architecture may be centralistic and/or equalized. In a centralistic system architecture a central server stores centrally the creations and activities of each individual client in a central log. In an equalized system architecture, each client stores the creations and activities of his/her own and other clients in a personal log. The clients personal log is updated during an update call to a central server performed during an update ring call to other clients. Update calls and update ring calls may be triggered by the client or automatically dependent on an available transmission capacity, or other definable parameters.
The invention and in particular the alternate embodiment may be applied to any communication system and particularly to a wireless communication system with inconsistent transmission capacities and arbitrary interruptions of connections.